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Abstract. A new necessary minimality condition for the Mumford-Shah functional is derived 
by means of second order variations. It is expressed in terms of a sign condition for a nonlocal 
quadratic form on Hq (T) , T being a submanifold of the regular part of the discontinuity set of 
the critical point. Two equivalent formulations are provided: one in terms of the first eigenvalue 
of a suitable compact operator, the other involving a sort of nonlocal capacity of T . A sufficient 
condition for minimality is also deduced. Finally, an explicit example is discussed, where a 
complete characterization of the domains where the second variation is nonnegative can be 
given. 
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1. Introduction 

The subject of this paper is the derivation and the analysis of a new minimality condition for 
the Mumford-Shah functional, obtained by means of second order variations. 

The homogeneous Mumford-Shah functional on a Lipschitz domain f2 in M. N , N > 2 , is 
defined as 

f(u,k)= [ \s7u\ 2 dx + n N ^ 1 {nnK), (1.1) 

Jn\K 

where Ti 1 ^' 1 is the [N — 1) -dimensional Hausdorff measure and (u,K) is any pair such that K 
is a closed subset of R N and u belongs to the Deny-Lions space L 1,2 (fl\K) (we refer to Section 2 
for the definition of this space). In the sequel the class of all such pairs will be denoted by A(£l) 
and its elements will be called admissible pairs. The functional (1-1), which was introduced in 
[12, 13] in the context of image segmentation problems, arises also in variational models for fracture 
mechanics (see [7] and [6]). 

Let (u,K) G A(Q) be a Dirichlet minimizer of F, that is, 

F{u,K)<F{v,K') (1.2) 

for every (v, K 1 ) 6 A(fl) with v = u on dil in the sense of traces. It is well known that u is 
harmonic in Cl\K and satisfies a Neumann condition on K; more precisely, u solves the equation 

(Au = Q inQ\K, 
1 d v u = on K. 

As for the regularity of the discontinuity set K , one can prove (see [2] and [3] ) that K fl O can 
be decomposed as 

Knn = T r ur s , (1.4) 

where T s is closed with H N ~ 1 (r s ) = and T r is an orientable (N ~ 1) -manifold of class C°° . 
Since u is of class C°° up to T r by (1.3), the traces Vi^ of Vu are well defined on both sides 
of T r . By considering variations of T r one can show (see [13]) that the minimality (1.2) implies 
also the following transmission condition: 

\Vu + \ 2 -\VuT\ 2 = H on IV, (1.5) 
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where H is the mean curvature of T r . We point out that by (1.3) and (1.5) the function u is 
subject to overdetermined boundary conditions on T r . Exploiting this observation it has been 
proved in [10] that T r has in fact analytic regularity. 

Let now (u, K) be a critical point of F, that is, a pair in A(Cl) satisfying (1.3), (1.4), and 
(1.5). Due to the nonconvexity of F one cannot expect these conditions to be in general sufficient 
for minimality. Nevertheless using a calibration method it is possible to prove that critical points 
are Dirichlet minimizcrs on small domains. More precisely, it has been proved in [11] that, if 
N = 2 , for every regular arc T compactly contained in T r there exists a tubular neighbourhood 
U of r such that (u,K) is a Dirichlet minimizcr of F in U. The minimality on large domains 
can fail in a rather surprising way: there might exist critical points whose energy can be strictly 
lowered by considering arbitrarily small diffeomorphic deformations of the regular part T r of K . 
An example of this phenomenon was given in [11, Proposition 4.1] by considering the critical point 
(u ,K ), where u (x,y) = x for y > 0, u (x,y) = —x for y < 0, and K = {y = 0} . If ft is the 
rectangle (xq,xq + £)x(— yo,Uo) with I and yo large enough, one can show that the functional 
can be decreased by perturbing T r — (xo,Xo + ^)x{0} by a diffeomorphism arbitrarily close to 
the identity. 

In this paper we begin a study of second order necessary conditions for minimality. More 
precisely, given a Dirichlet minimizer (it, K) , we compute the second derivative of the energy 
along variations of the form (u E , K £ ) , where K E = $ e (K (~) 0) , $ e being a one-parameter family 
of diffcomorphisms coinciding with the identity on a fixed neighbourhood of T s U <9f2 , and u e is 
the solution of the problem 



muJ / \Vv\ 2 dx: v E L 1 ' 2 (n\K e ), v = u on <9Q 



N v \2i- . .. rl,2/ 

In 

This approach has some similarities with the computation of "shape derivatives" introduced in 
[15] in the context of shape optimization problems. As <I> e coincides with the identity on a 
neighbourhood of T s , the singular part of the discontinuity set is left unchanged by the variation, 
which can thus affect only the regular part T r . We also point out that the variation u £ of the 
function u has a nonlocal character. This is crucial to retrieve information about global properties, 
such as the size and the geometry of fi and T r . 

Whereas the first order variation of F along (u e ,K £ ) gives back the equilibrium condition 
(1.5), the second order variation provides us with a new necessary minimality condition, expressed 
in terms of a sign condition for a quadratic form depending on (u, K) . More precisely, for every 
submanifold T compactly contained in T r and every Lipschitz domain U C £1 we consider the 
functional S 2 F((u, T); U) on H&(TnU) defined as 



S 2 F((u,T);UM :=2 / (v+d„v+ - v~d v v~) dH N ~ x + [ \V r v\ 2 dH N - 
Jvnu Jrnu 

+ [ (2B[V r w + , V r w + ] - 2B[V r u~, V r u~] - \B\ 2 )(p 2 dH N ' ' 
Jrnu 

for every ip £ Hq {T (~l U) , where v v G i 1,2 (?7\i4r) solves the problem 

'Av v =Q mU\K, 

v v = on dU, 

d u v± = div r (^V r u ± ) on T n U, 

,d v v v = onKnU\T. 

Here the symbols Vr and divr denote the tangential gradient and the tangential divergence on 
r , B is the second fundamental form of T , while denote the traces of v v on the two sides 
of r. As v v depends linearly on tp, the functional S 2 F((u, T); U) defines a quadratic form on 

We first show (Theorem 3.15) that, if (u,K) is a Dirichlet minimizer of F, then for every T 
and U as above we have the second order condition 

S 2 F((u,T);U)[(p] > for every (peH^(TnU). (1.6) 
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Conversely, we prove (Theorem 5.1) that, if TV < 3 and (u, K) is a critical point satisfying the 
stronger condition 

5 2 F{{u,r);U)[<p} > for every <p e H^{TnU)\{0} 7 (1.7) 

then (it, K) is a minimizcr of F on U with respect to all pairs (v, <fr(K n U)) such that <I> is a 
diffcomorphism belonging to a C 2 -neighbourhood of the identity and coinciding with the identity 
on KH U\T, and v G L 1 > 2 ([/\$(.K'nt/)) with » = !ion3l/. The restriction N < 3 is a technical 
assumption. In fact, a slightly weaker minimality property is shown to hold in any dimension (see 
Remark 5.7). 

A detailed study of the stronger condition (1.7) is performed in Section 4, where two equivalent 
formulations are shown. The first one (Theorem 4.6) is a condition on the first eigenvalue of the 
(nonlocal) compact operator T : H^(T n U) — > H^(T n U) , defined for every ip e i?o( r n ^) as 

Tip = R(2\7 r u+ ■ \7 r v+ - 2X7 rU - ■ V r v~). 

Here R : i3" _1 (r n U) — > i?o ( r n ^) denotes the resolvent operator which maps / e i? _1 (r n ?7) 
into the solution 9 of the problem 

j-ArO + ae = f inrnC/, 

lfle^(rnc/), 

where Ar is the Laplace-Beltrami operator on T and 

a(x) = 2B(x)[V r u + (x),V r u + (x)} - 2B(x)[V r w~(x), W r u-(x)} - \B(x)\ 2 . 
The second equivalent formulation (Theorem 4.10) is expressed in terms of the variational problem 

min{2 f \Vv\ 2 dx : v E l}' 2 (U\K), v = on dU, [ {aift + | VrV«| 2 ) dH 1 "' 1 = l), 

where ip v = i?(2Vru + • Vr« + — 2Vru~ • Vr«~) • This minimum problem describes a sort of 
nonlocal "capacity" of T with respect to U , where the usual pointwise constraint v = 1 a.e. on 
r n U is replaced by the integral condition on ip v . We also note that this second formulation is 
strictly related to the sufficient condition for graph- minimality studied in [11]. It is easy to see that 
the sufficient condition in [11] is stronger than (1.7) and in fact it implies a stronger minimality 
property. The comparison between the two conditions is discussed in the explicit example of 
Section 7 (see Remark 7.2), where we consider the critical point (uo,Kq) of [11, Proposition 4.1] 
and we give a complete characterization of the rectangles U = (x ,x + £)x(—y ,y ) where 
condition (1.7) is satisfied. 

Finally, we prove some stability and instability results. We first show that, if (u,K) is a 
critical point, then condition (1.7) is automatically satisfied when the domain or the support of 
the variation is sufficiently small (Propositions 6.1 and 6.3). Instead, condition (1.6) may fail if 
the domain is too large (Proposition 6.5). This is in agreement with the two dimensional results 
of [11]. 

It remains an open problem to understand whether condition (1.7) implies a stronger minimality 
property, in analogy to the classical results of the Calculus of Variations for weak minimizers. This 
would probably require the use of different techniques, such as calibration methods or Weierstrass 
fields theory. 

It is our intention to investigate variations involving also the singular part T s of the discontinuity 
set in future work. Moreover, it is our belief that the techniques developed in this paper can be 
applied to more general functional, both in the bulk and in the surface energy. 

The plan of the paper is the following. In Section 2 we collect all the notation and the pre- 
liminary results needed in the paper. Section 3 is devoted to the derivation of the second order 
necessary condition (1.6). In Section 4 we discuss the equivalent formulations of the sufficient 
condition (1.7), which is proved in Section 5. Stability and instability results are the subject of 
Section 6, while the explicit example in dimension 2 is studied in Section 7. Finally, the regularity 
results needed in the derivation argument are collected and proved in Section 8. 
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2. Notation and preliminaries 

In this section we fix the notation and we recall some preliminary results. 

Matrices and linear operators. Given a linear operator A : R N — > R d , we denote the action of 
A on the generic vector h £ M. N by A[h] . We will usually identify linear operators with matrices. 
We denote the cuclidean norm of a linear operator (or a matrix) A by 

\A\ := (trace (A T A)) 1/2 , 

where A T : R d — > stands for the adjoint operator. If d = N we can consider the bilinear form 
associated with A 

A[h 1 ,h 2 ]:=A[h 1 ]-h 2 for h 1 ,h 2 £ K w , 

where the dot denotes the scalar product of ~R N . Conversely, to any bilinear form B : R^ x ~R N — > 
R we can naturally associate a linear operator, still denoted by B, whose action on the generic 
vector h £ R w can be described by duality as 

B[h] ■ z = B[h, z] for every z £ R N . 

We will usually identify bilinear forms with the associated linear operators. 

Geometric preliminaries. Let Y C l w be a smooth orientable (N — 1) -dimensional manifold 
and assume that there exists a smooth orientable (N — 1) -dimensional manifold M such that 
r CC M . For every x £ Y we denote the tangent space and the normal space to Y at x by X^r 
and N X Y , respectively. 

Let S N ^ X be the (N — 1) -dimensional unit sphere in R N . We call an orientation for Y any 
smooth vector field v : T — > such that v{x) £ N X Y for every x £ Y . Given an orientation 

we can define a signed distance function from Y, which turns out to be smooth in a tubular 
neighbourhood hi of Y and whose gradient coincides with v on Y . The extension of the normal 
vector field provided by the gradient of the signed distance function will be still denoted by 

We now recall the definition of some tangential differential operators. Let g : U — > M. d be a 
smooth function. The tangential differential drg(x) of g at x £ T is the linear operator from R N 
into R d given by drg(x) := dg(x) o n x , where dg(x) is the usual differential of g at x and tt x is 
the orthogonal projection on T X Y . We denote the matrix (the vector if d = 1) associated with 
drg{x) by Drg(x) (\7 r g(x) if d = 1). As remarked above we will often identify matrices with 
linear operators. Note that 

(D T g(x)) T [h]-v(x) = h-D T g(x)[v(x)] = for every h £ R d , 

that is, (Drg{x)) T maps R d into T X Y . We remark also that by our choice of the extension of v 
around Y we have 

Dv = D T v onT. (2.1) 
If d = N we can define the tangential divergence divrg of g as 

JV 

div r 5 : =^2 e j - ^r9j, 

where ei, . . . , ejv are the vectors of the canonical basis of R.^ and <7i, . . . , are the corresponding 
components of g. It turns out that 

N-l 

div r g{x) = T J- d T 3 9> 
j'=i 

where ti, . . . ,tn-i is any orthonormal basis of X^r and for every v £ the symbol d v 

denotes the derivative in the direction v . Sometimes it is also useful to bear in mind the identity 

div g = divrg + v • d v g. 
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In particular, as d v v — by (2.1), we deduce that A\nv = div r ^ on T. We will make repeated 
use of the following identities: 

div r (<A9i) = Vrp-Si + ¥>divr5i, 

Vr(ffi -92) = {D T gi) T [g2\ + (£>rg 2 ) T [gi]. 

for (p £ C X {}A) and (71,(72 € C 1 ^; IR^) . Finally, we recall that the Laplace- Beltrami operator Ar 
on T is defined as 

A r g := divr(Vrg) 

for every smooth real valued function g . We remark that all the tangential differential operators 
introduced so far have an intrisic meaning, since they only depend on the restriction of g to T . 
For every x £ T we set 

BO) := D T v{x) = Dv{x). (2.2) 
The bilinear form associated with B(x) is symmetric and, when restricted to T x TxT x T , it coin- 
cides with the second fundamental form of T at x. It is also possible to prove that T X T is an 
invariant space for B(x) . 

We consider also the function H :U — > R defined by 

H:=&ivv. (2.3) 

On T we have H = divf = divr^ = trace B, that is, for every x £ T the value H{x) coincides 
with the mean curvature of T at x . 

It is important to recall the following divergence formula: 

J divrgdH"- 1 = J H(g-is)dH N -\ (2.4) 

which holds for every smooth function g :U — > l w with suppg n V CC T . Note that (2.4) allows 
to extend to tangential operators the usual integration by parts formula. Indeed, we have 

J ipdivrgdH 1 *- 1 = - J Vrcp-gdH"- 1 (2.5) 

for every smooth g :U — > R N such that g{x) £ T X T for x £ T , and every smooth ip :U — > 1R with 

supp^nrccr. 

Let U be a bounded open set in R N with U C\Y 7^ and let $ : U — > U be a smooth orientation- 
preserving diffeomorphism. Then T$ := <f>(m U) is still an orientable smooth (N — l)-manifold. 
A possible choice for the orientation is given by the vector field 

Accordingly we can define the functions B$ and H$ as in (2.2) and (2.3), with T and v replaced 
by r$ and v<§ , respectively. We shall use the following identity, which is a particular case of the 
so-called generalized area formula (see, e.g., [2, Theorem 2.91]): for every ip £ L x (r$) 

/ i>dH N - 1 = f {il>o$)J^dH N -\ (2.7) 

where J$ := |(D$)~ T [^]| det D<I> is the (N - 1) -dimensional Jacobian of $. 

We conclude this subsection by introducing the Sobolev space Hq (T) , which is defined as the 
closure of C^°(T) with respect to the norm 



MIW) := J r (\u\ 2 + \Vru\ 2 )dH N ~ : 



Many of the properties of classical Sobolev spaces, such as Poincare inequalities and integration 
by parts formulas continue to hold. We refer to [9] for a complete treatment of these spaces. We 
shall denote the dual space of Hq(T) by i7 _1 (r). 

Deny-Lions spaces. Given a bounded open subset £1 C M. N , we say that fl has a Lipschitz 
boundary at a point x £ d£l if there exist an orthogonal coordinate system (yi, . . . , t/jv) , a 
coordinate rectangle R — (ai, b\)x . . . x (ajv, &jv) containing x, and a Lipschitz function * : 
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(ai,6i)x . . . x(ojv_i,&jv-i) -» {a.N,b N ) such that ftni? = {y £ R : y N < . . .,y N -i)}- The 

set of all such points, which is by definition relatively open, is denoted by <9t,Q. If d[£l = d£l we 
say that fl is a Lipschitz domain. 

To deal with possibly unbounded functions in problem (1.3), besides the classical Sobolev space 
if 1 (ft) we shall also use the Deny-Lions space 

L l < 2 {Q) := {u £ L 2 oc (ft) : V« £ L 2 (ft; M. N )}, 

which coincides with the space of all distributions on ft whose gradient belongs to L 2 (tt;M. N ) . 
In the brief account below we essentially follow [4, Section 2] (see also [5]). The relation between 
Sobolev and Deny-Lions spaces is unveiled by the following proposition. 

Proposition 2.1. Let u £ L 12 (Q) and let x £ 9 L ft. Then there exists a neighbourhood U of x 
such that u\n n u £ i? 1 (fin U) . In particular, if ft is Lipschitz, then L 1,2 (ft) = iJ 1 (fi). 

Let A and B be 7i w_1 -measurable sets in M. N . We say that A is quasi- contained in B, and 
we write A£ B , if H N ~ 1 (B\A) = 0. It is known that every function in L 1,2 (ft) can be specified 
at H" _1 -a.e. point of ft U d^ft . Hence, if A C <9ft is relatively open and Ac c^ft, we can define 
the space 

£j' 2 (ft; A) := {u £ Z, 1>2 (ft) : u = H" _1 -a.e. on A}, (2.8) 

where we identify functions which differ by a constant on the connected components of ft whose 
boundary docs not meet A. With this identification, arguing as in [4, Corollary 2.3], one can 
prove the following. 

Proposition 2.2. The space Lq' 2 (£1;A) introduced in (2.8) is a Hilbert space endowed with the 
norm || Vu\\ L 2 {n . MN) . 

3. The second variation 

In this section we define and compute a suitable notion of second variation for the Mumford- 
Shah functional (1.1). We recall that A(Cl) is the class of all pairs (u, K) such that if is a closed 
subset of and u £ L 1 ' 2 {£l\K) . It is useful to "localize" the definition of F to any open subset 
U C f2 by setting 

F((u,K);U):= [ |Vw| 2 dx + H N -\U n K) 

JU\K 

for every admissible pair (u, K) £ A{Q) . 

In the sequel we shall consider only admissible pairs (u, K) which are partially regular in the 
sense of the following definition. 

Definition 3.1. Let fl C M w be a Lipschitz domain and let (u, K) £ A(Vl) . We say that (u, K) 
is partially regular in O if dfl £ 8l(^\K) (see the end of Section 2), u solves the problem 

mini/ \\7v\ 2 dx: v-u£L^ 2 (n\K;dn)\, (3.1) 
Jn\K > 

and K can be decomposed as K = r r ur s , with r r nr s = 0, T s relatively closed, H. N ~ 1 (T S ) = 0, 
and r r orientable (N — 1) -manifold of class C°° . We denote the class of all such pairs by 
A re g(Q) ■ Finally we say that U C O is an admissible subdomain for (u,K) if it is Lipschitz and 

(u,KnU)£Areg(U). 

Remark 3.2. The previous definition is motivated by the regularity results for local minimizers 
of free discontinuity problems (see [2] and [3]). Note also that u solves (3.1) if and only if 

[\7u-\7zdx = for every z £ Ll' 2 (n\K;dn), (3.2) 
Jq 

where we also used the fact that K has Lebesgue measure equal to 0. 

In the next definition we introduce the class of admissible variations of the discontinuity set K . 
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Definition 3.3. Let il C l w be a Lipschitz domain, let (u,K) G A reg (fi) , let T CC T r be 
relatively open, and let U C fl be an admissible subdomain for (u, K) according to Definition 3.1. 
We say that ($t)te(-i,i) is an admissible flow for Y in U if the following properties are satisfied: 

(i) the map (t,x) ^ ® t (x) belongs to C°°((-l, l)x77;?7) ; 

(ii) for every t e (-1, 1) the map <I> t is a diffeomorphism from U onto itself; 
(hi) $o coincides with the identity map I in U; 

(iv) there exists a compact set G C U \ (K \ T) such that supp ($ t — I) C G for every 

te(-i,i). 

Remark 3.4. Condition (iv) in the previous definition implies that $t can affect T only, while 
(Kr\U)\T remains unchanged. We also remark that from the assumptions T has positive distance 
from T s U dtt , where singular behaviour of the function u can occur. 

Finally, we describe the variation of u associated with an admissible variation of its discontinuity 
set K. Let VI, (u,K), and U be as in Definition 3.3. Given a diffeomorphism $ G C°°(U;U), 
satisfying condition (iv) (with $ t replaced by <&), we define u$ as the (unique) solution of 

' u<s> — u G Ll' 2 (U\K<s>; dU), 



1,2,™ „ nrn ( 3 - 3 ) 



Vucj, -Vzdx = for every z G L ' (U\K$;dU), 

u 

where u:— (pu and ^ is a cut-off function such that (p = on G and <£> = 1 in a neighbourhood 
of dU . In particular, u^> — u TL N ~ 1 -a.e. on <9£7. 

We are now ready to define our notion of second variation. 

Definition 3.5. Let ft, (u,K), U, T, and (<j> t ) be as in Definition 3.3. We define the second 
variation of F at (u, K) in U along the flow ($ t ) to be the value of 

^F((«* t ,^* t );^)| t=0 , (3.4) 

where K<j, t := $ f (i(n U) and u$ t is defined by (3.3) with $ replaced by $ t . 

We point out that the existence of the derivative (3.4) is guaranteed by the regularity results 
of Section 8. 

We fix now some notation which will be repeatedly used in the following discussion. For any 
one-parameter family of function (<? s ) s e(-i,i) the symbol gt{x) will denote the partial derivative 
with respect to s of the map (s,x) g s (x) evaluated at (t,x). To be more specific, let 0, 
(u,K), U, T, and ($ t ) be as in the previous definition. For every t G (—1, 1) we set 

:= o $r\ Zs> •= o $r\ 



where, according to the previous notation, 

d ■■ d 2 

Similarly, for every t G (—1,1) we define u$ t as the partial derivative with respect to s of the 
map (s,x) i— > u$ a (x) evaluated at (i, x). Proposition 8.1 in the appendix guarantees that the 
derivative exists and that u$ t G L^ 2 (U\K$, t ; dU) . We shall often omit the subscript when t = 0; 
in particular, we set 

ii := u$ , X := X$ , Z := .Z$ . (3-5) 

We define X" as the orthogonal projection of X onto the tangent space to T, that is, X^ := 
(7—i/® . Finally, for any function z G L 1,2 (Q\K) we denote the traces of z on the two sides 
of T by z + and z~ . More precisely, for H N ~ 1 -&.e. x G T we set 



where C N is the iV -dimensional Lebesgue measure, B r {x) is the open ball of radius r centered 
at x, and V± := {y eR N : ±(y - x) ■ v{x) > 0}. 
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In the next theorem, which is the main result of the section, we compute the second variation of 
F , according to Definition 3.5. We refer to Section 2 for the definition of all geometrical quantities 
appearing in the statement. 

Theorem 3.6. Let 0, (u,K), U , T, and ($ t ) be as in Definition 3.3. Then the function u 
belongs to L^ 2 {U\K;dU) and satisfies the equation 

I Vu-Vzdx+ I [div r ((X ■v)X7 r u + )z + -div r {{X ■v)\7 rU -)z-]dH N - 1 = (3.6) 
Ju Jmu 

for all z e L ' (U\K;dU) . Moreover, the second variation of F at (u,K) in U along the flow 
(<&t) is given by 

^F((u* t ,K* t );U)\t=o = 2 / (u+d u u+ - iTduiT) dH 1 ^- 1 + f \ V r (X ■ u)\ 2 alU*- 1 
dt Jmu Jmu 

+ [ (X ■v) 2 (2B[\7 r u + ,\7 r u+}-2B[\/ r u-,\7 r u-}-\B\ 2 )dH N - 1 (3.7) 
Jmu 

+ f f(Z-v-2XW -V r (X -u)+B[X\xH] + H(X ■v) 2 )dH N - 1 , 
Jmu 

where f := |V r w~| 2 - |V r u+| 2 + H. 

Remark 3.7. The first part of the previous theorem implies that ii is harmonic in U\K , u = 
on dU , 9^u ± = divr((X • i/)Vr?i ± ) on TtlU, and u satisfies a weak homogeneous Neumann 
condition on K n U\T . In particular, using u as a test function in (3.6), we have 



/ {u + d u u + - u~d v u-) dH N ^ = - [ |V«| 
Jmu Ju 



2 dx. 



imu Ju 
The following lemma contains some useful identities, which will be repeatedly used in the proof 
of Theorem 3.6. The proof of the lemma is postponed until Section 8. 

Lemma 3.8. The following identities are satisfied on T : 

(a) V 2 u ± [j/, v] = -Aru*; 

(b) VV^A",!/] = -(X-z^Aru* -BtVru^-X"]; 

(c) div r [(X • i/JVru*] = {D T X) T [v, V r u±] - X7 2 u ± [X, v\ ; 

(d) d v H = -\B\ 2 ; 

(c) V 2 « ± hV r « ± ] = -BfVru*, Vr^]; 

(f) v = -{D T X) T [v}- D T v[X]; 

(g) ^ ($ t • (i/* t o $ t ) J* t )| t=0 = Z-v- 2Xll • V r (X • i/) + B[Xll,X"] + div r ((X • v)X) . 

We will also need the following well-known result on the first variation of the area functional 
(for the definition of T$ t and i?$ t we refer to Section 2). 

Proposition 3.9 (see [14]). The first variation for the area functional is given by 



±H N - 1 (r* t ) = £ H 9t {x^.v^)dn N -\ (3.e 



/r * t 

We are now in a position to prove Theorem 3.6. 
Proof of Theorem 3. 6. We split the proof into three steps. 

Step 1. Derivation of the equation solved by ii. By Proposition 8.1 we have that u £ Lq' 2 (U\ 
K;dU). Let z <G ij' 2 (U\K '; dU) with suppznT = 0. Then, suppz C U\K^ t for t small enough, 
so that, in particular, z e Lq' 2 ''(U\K$ t ;dU) . Hence, by (3.3) we have /pVtij, • Vz dx = 0. 
Differentiating with respect to t, we deduce 



Vu-Vzdx = for every z e L L ' 2 (U\K;dU) with suppzHT = 0. (3.9) 

u 
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Note that by (2.6) one has 

^°^ = \mpR onr ' (3 ' 10) 

It is convenient to set w t := (D§ t )~ T [v\ (as usual, we shall omit the subscript t when t = 0). As 
^* t M $ t = on T$ t by (3.3), we have (VuJ t o $ t ) • (y< s>t o <£> t ) = and in turn, using (3.10), 

(Vu± o $ t ) • Wt = onT. (3.11) 

Differentiating (3.11) with respect to f at f = and using the fact that w = — [DX) T [v] on T , 
we obtain 

dvi^ = -V^psT,;/] + (L»X) t [^,Vm ± ] = -VV=[A>] + (L>r^) T [^,V r u ± ], 

where in the last equality we used that Vi^ = Vru"* 1 on T . By (c) of Lemma 3.8 we conclude 
that 

=div T [(X-v)W r u ± ] onT. (3.12) 

Now let z e Ll' 2 (U\K;dU) . We can write z = zi+z 2 , where suppzi CC U and suppzinr s = 
0, while suppz 2 n f = 0. Then, by (3.9) and (3.12) we finally obtain 



N-l 



/ Vu-Vzdx^ I Vii-Vz 1 dx= / [div r ((X • v)\7 r u~)z- - div r ((X • v)V T u + )z + ] 
Ju Ju Jmu 

where the last equality follows by integration by parts. This establishes the first part of the 
statement. 

Step 2. Computation of the first variation. We shall show that 

j t F{{u* t ,K* t );U) = jT (|Vr. tU$ J 2 - |V r * t < I' + H* t ){X* t ■ i/*J dU^ 1 (3.13) 

for every t € (—1, 1) . 

We start by performing a change of variables in the integral, which leads to 

J \Vu^ t \ 2 dy = J \\7 u<s>t o$ t | 2 det.D$ t dx = \\y/detD¥ t (Vu* t ° ®t)\\ 2 L 2 {u . R wy 

By the regularity results of Proposition 8.1 and by the identity 

d 

— (det£>$ t ) = (divXj, t o$ t )detD$ t 
(see [8, Chapter III, Section 10] for a proof), we obtain 

= 2 J v/det£>$ t (Vw$ t o $ t ) • — (VdetD* t (Vu* t o $ t )) da; 

= / |Vu$ t o$ t | 2 (divX$ t o$ t )detL>$ t dr 
J(7 



'(7 

2 / (Vtt$ t o$ t )-(Vu* t o$ t )detD$ t dx 
Ju 



+ 2 (V 2 u$ t o $ t )[Vu$ t o $ t ,$ t ] detL»$ t dx 

= / |Vu$ t | 2 divJf$ t dy + 2 / V?i$ t • Vu$ t dy + 2 / V 2 w$ t [Vu$ t , Xj, t ] dy 
J(7 •/[/ •/[/ 

= / div(|Vu$ t \ 2 X$ t ) dy + 2 / Vu$ t • Vu$ t dy = / div(| Vw$ t | 2 X$ t ) dy, 
J(7 Ju Ju 
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where the last equality follows from (3.3), since u$ t e L^' 2 '(U\K< s , f ; dU) . Integrating by parts we 
deduce 

| J u \Vu^\ 2 dy = ^ (IVr^^r-IVr^n+l 2 )^.^)^" 1 , 
which, together with (3.8), gives (3.13). 

Step 3. Computation of the second variation. We are now ready to compute (3.7). To simplify 
the notation in the calculations below we set f t := | | 2 — |VttJ | 2 + H$ t . Using the fact that 
|Vr* t w$ | 2 = |Vu$ | 2 on T$ t , the area formula (2.7), and the identity X$ t o$ f = $ t ,we find 

_F((n $t ,^J;C/)| t=0 = -( ^ / t (X* t • i/*J dW"" 1 ) 



/ ^-Aft ° $t)\t=o(X ■ v) dH N - 1 + [ f%-(*f(v* t o* t )J* t )\ t=0 dH I 
Jmu ot Jmu ot 



t=o 



=: /i+J 2 . (3.14) 
The first integral 1\ can be written as 

h = f fiX-^dH"- 1 + f {Vf-X){X-v)dU N - 1 
Jmu Jmu 

= f f{X-v)dH N - 1 + f {Vf-v){X-vfdH N - 1 
Jmu Jmu 

+ f {Vrf-X^){X-v)dH N - 1 . (3.15) 
Jmu 



imu 

By property (g) of Lemma 3.8 the second integral Ii turns out to be 

I 2 = [ f(Z-v-2XW ■ V r (X ■v) + B[x\xU})dH N - 1 + f fdiv r ((X ■ v)X) dH N -\ (3.16) 
Jmu Jmu 

We note that by (2.4) we have 

/ fdiv r ((X-iy)X)dH N - 1 + ( {V T f-XW){X-v)dH N - 1 
Jmu Jmu 

= f div r (f(X-is)X)dH N - 1 = f !H{X- v fdU N - x . 
Jmu Jmu 

Combining the previous identity with (3.14)-(3.16) we obtain 

f^F((u^,K^);U)\ t= o = f f{Z.v-2X\\.V T {X-v)+n[X\\,X\\}+H{X.vY)dH N - 1 
at Jmu 

+ f f(X-^)dH N - 1 + f {Vf-v){X-ufdH N -\ (3.17) 
Jmu Jmu 

Using the definition of / and properties (d) and (c) of Lemma 3.8, the last term in the previous 

expression can be written as 



/ {Vf-v)(X-vfdH N -' 
Jmu 

= [ {2V 2 u-[v,V T u-] -2V 2 u + [^,V r w + ] +d u H){X-v) 2 dH N - 
Jmu 

= [ (2B[V r u + ,V r u + }-2B[V r u-,V r u-}-\B\ 2 )(X-iy) 2 dn N - 1 . (3.18) 
Jmu 

Differentiating / with respect to t, we obtain 

f f(X-v)dH N - 1 = [ (2V r w~ -Vrii- - 2V r u+ • V r u+ + H){X ■ v) dH N ~ x . (3.19) 
Jmu Jmu 
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Integrating by parts, according to (2.5), and using (3.6) (see also Remark 3.7), we deduce 



if (Vr^-Vru^iX-^dH 1 *- 1 = -2 f w ± div r ((X • i/JVru*) dH N ~ 
Jmu Jmu 

-,±« „-.± jn-iN-l 



= -2 / ^d^dH"- 1 . (3.20) 
Jmu 

Since d v v-v = —i>-d v v = by (2.1), we have divj> = divpi' and in turn, by (2.3), H = divpi'. 
Hence, integrating by parts and using (f) of Lemma 3.8, we deduce 

f H{X-v)dH N - 1 = f div r *> (X ■ v) dU N ~ x = - f v -V r (X ■ v) dU N ~ x 
Jmu Jmu Jmu 

= f {{D r X) T [v\ + D r u[X})- V T {X-v)dH N - 1 
Jmu 

= f IVr^-^l 2 ^" 1 . (3.21) 
Jmu 

Combining (3.17)-(3.21), we obtain (3.7) and we conclude the proof of the theorem. □ 
Remark 3.10. Let us fix s e (—1, 1) . We observe that the family of diffeomorphisms 

$ h := $ s+h o 

is an admissible flow for T$ s = 3> s (rn [/) in U (one can always reparameterize the "time" 
variable h away from so that is defined for all h G (—1, 1)) and that ($h)\h=o — X^ s and 
{&h)\h=o — Z<& 3 ■ Applying Theorem 3.6, we deduce that 

±^F((u^,K^);U)\ t=s = ■^F((u 9 . +h ,* h (K* a )) ] U)\ h=0 



= if (u+d v „u+. -u*d v * a u* s )dH N - 1 + f IVr^P^.^JI 2 ^" 1 

+ f (X $s -^J 2 (2B$ 5 [Vr 4s n+ , V r .,u+ ] - 2B $s [V r * s % , V r * 5 % ] - |B$J 2 ) dH 1 "- 1 

+ / f,(Z*. ■ m 3 - 2X1 ■ Vr* s (X* s ■!/».) + B* s [xl s ,xl 3 ] + H* 3 (X* s ■ v* J 2 ) dH N ~\ 
Jr^ s 

where f s := |VuJJ 2 — |VuJJ 2 + i?$ s . Moreover, u$ s belongs to L ' 2 (U\K<j, s ; dU) and satisfies 

/ Vii$ a • Vz dx 
Ju 

+ / (div r , 3 ((X $3 .^jVr, 3 4 s )z + -div r , 3 ((X $3 -^jVr^^Jz-)^- 1 =0 
Jr^ s 

for every z e L^ 2 {U\K^ 3 ;dU) . 

As already explained in the introduction, in the context of this paper critical points are partially 
regular admissible pairs which satisfy an additional transmission condition along the discontinuity 
set. 

Definition 3.11. Let f2, (u,K), U, and L be as in Definition 3.3. We say that (u, K) is a 
critical point in U with respect to T if 

H = |V r w + | 2 - |V r u~| 2 onTnt/. (3.22) 

If (u, if) is a critical point, then the expression of the second variation of F at (u, K) simplifies, 
as the function / vanishes. We have therefore the following corollary. 
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Corollary 3.12. In addition to the hypotheses of Theorem 3.6 assume that (u,K) is a critical 
point in U with respect to T . Then 



^F((u^,K^);U)\ t=0 = 2 f (u+d„u+ -u-d^dH"- 1 + [ \V T {X-v)\ 2 t 
dt Jmu Jmu 

+ [ (X ■v) 2 (2B[Vru + ,Vru + }-2B{V r u-,V r u-}-\B\ 2 )dH I 
Jmu 



<mu Jmu 

/N-l 

v\ ■ v ) ^-D[ vr'u | vr'u J — m[ v pa ) vr'u J ™ 1-°! )un 

>mu ' 

Remark 3.13. We note that, if (u, A) is a critical point in U with respect to L, then the second 
variation of (u,K) in U with respect to the flow ($ t ) depends only on the normal component of 
the variation X ■ v . Moreover, as u depends linearly on X ■ v , the second variation becomes in 
this case a quadratic form in the variable X ■ v . 

The previous corollary suggests the following definition. Given f2, (u,K), U, and T as in 
Definition 3.3, we can consider the function S 2 F((u, T); U) : Hq (T n U) — > M given by 



S 2 F((u,T);UM :=2 f (v+d„v+ - v~d v v~) dH^ 1 + f \V r ^\ 2 dH N ^ 
Jmu Jmu 

+ f (2B[VrM + ,Vrw + ] -2B[VrM",V r u-] - IBI 2 )^ 2 ^" 1 , (3.23) 
Jmu 

where v v £ L\ ,2 (U\K\dU) is the solution of 

I \7v v -\7zdx+ I (div r (ip\7 r u+)z+ -div r (<p\7 rU -)z-)dH N - 1 = (3.24) 
Ju Jmu 

for all z e Ll' 2 (U\K;dU) . As v v depends linearly on ip, the function 6 2 F((u, V); U) defines a 
quadratic form on Hq(T n U) . Arguing as in Remark 3.7, it is easy to see that 

d v v± = div r ((^Vru ± ) on Y n U (3.25) 



(v+d„v+-v-d v v-)dH N - 1 = - / \Vv v \ 2 dx. (3.26) 



and 

/ {v^pdv'J^p < } " < < - / | \ < _ 

Jrnt/ •/[/ 
We conclude this section by proving a second order necessary condition for minimality, expressed 
in terms of the quadratic form defined in (3.23). Minimality is intended in the sense of the following 
definition. 

Definition 3.14. Let 0, (u, A) , U, and T be as in Definition 3.3 and let k € NU {oo}. We say 
that (it, A) is a C k -local minimizer in U with respect to T if there exists 5 > such that 

f IVufdx + H^iKnU) < f iVvfdx + ^-^QiKCMI)) (3.27) 
Ju Ju 

for every C k -diffeomorphism $ on U with <I> = I on (AT n f/)\r and ||$ — I\\ci° < 5, and every 
v e L^ 2 (U\^(KnU)) withu = u H^^-a.e. on <9C/. We say that (u, A) is an isolated C k -local 
minimizer in U with respect to T if (3.27) holds with the strict inequality for every $ as before, 

with r$^rnc/. 

Every C°° -local minimizer has nonnegative second variation, as made precise by the following 
proposition. 

Theorem 3.15. Assume that (u,K) is a C°° -local minimizer in U with respect to T. Then the 
quadratic form (3.23) is positive semidefinite; i.e., 

8 2 F{{u, T); U)[ip] > for every tp G H&(T n U). (3.28) 

Proof. Let us fix ip € C£°(T n ?7) and consider an admissible flow (<j> t ) for L in [/ such that for i 
small enough <f> t = (/ + tipv) oH r in a neighbourhood of T n [/ , where ITr denotes the orthogonal 
projection on T. It turns out that the vector field X , introduced in (3.5), coincides with pv on 
m U . Using Corollary 3.12 and the minimality of (u, K) we then deduce 

6 2 F((u,T);UM = ^F((«* t ,tf« t );[0| t=0 > 0. 
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The thesis follows by approximating any ip £ Hq(T CiU) with functions in C^°(T (lU). □ 

4. Equivalent formulations of the second order condition 

Throughout the whole section (u, K) will be a pair in *4 re9 (f2) and U C £1 an admissible 
subdomain for (u,K) in the sense of Definition 3.1, while T will denote a relatively open set 
compactly contained in T r . 

The purpose of this section is to perform a more detailed study of the second variation. In 
particular we shall derive some necessary and sufficient conditions for the second variation to be 
positive definite in admissible subdomains U of O; i.e., 

S 2 F ((u,T);U)[(f] > forevery p e H^(T D U)\{0} . (4.1) 

In the first subsection we show that (4.1) is equivalent to a condition on the first eigenvalue 
of a suitable compact operator T on Hq (T CiU). In the second subsection we formulate (4.1) in 
terms of a dual minimum problem. 

4.1. An equivalent eigenvalue problem. We introduce a bilinear form on Hq(T n U) defined 

by 

(p,V)~:=/ a<pijam N - l +( \7 r p ■ \7 r ^ dH N ^ (4.2) 
Jrnu Jrnu 

for every p,ip e Hq(T (~l U) , where 

a(x) := 2B{x)[V r u + (x),V r u + (x)} - 2B{x)[V r u- (x),V rU - (x)} - \B(x)\ 2 
for every x £ T (~l U . 

Remark 4.1. All the results contained in this section do not depend on the special form of a 
and continue to hold whenever a is replaced by any smooth bounded function on T . 

We start by showing that the bilinear form (4.2), when it is a scalar product, is indeed equivalent 
to the standard scalar product of Hq (T D U) . 

Proposition 4.2. Assume that 

(ip,<p)^>0 for every ip G H*(T nU)\{0} . (4.3) 

Then the bilinear form (4.2) defines an equivalent scalar product on Hq(T n U) . 

Proof. Assumption (4.3) immediately implies that the bilinear form (4.2) is a scalar product. In 
particular, 

|MU:=(^)V 2 (4.4) 

defines a norm on Hq(T (~l U) . 

To show the equivalence with the scalar product of Hq(T f~\U), we first observe that, as a is 
bounded, we have ||<p||~ < C||</?||#i for every ip e Hq(T (III) . For the opposite inequality we 
argue by contradiction assuming that there exists a sequence (ip n ) such that ||</?„|| ff i = 1 and 

M~ < I (4-5) 

Then, up to subsequences, ip n ~± ip weakly in Hq(T CiU). In particular, tp n — > tp in L 2 (T n U), 
hence 

jN-l _ ,■ / „, 2 JnjN-1 



[ atp 2 dH N - 1 = lim f atp 2 n dU 
Jrnu n Jrnu 

f iVr^l 2 ^- 1 < liminf f \\7 r p n \ 2 dH 
Jrnu n Jrnu 



: i- / " Jrnu ( 4 _ 6 ) 

I"- 1 < liminf [ I* 7 — i 2 ^" 1 
imu n Jrnu 

Recalling (4.5) it follows that \\(p\\^ = 0, that is, tp = 0. Using again (4.5) and (4.6), we deduce 

that J Tnu |Vr<^n| 2 dTL 1 ^^ 1 — ► 0, which contradicts the fact that ||<Pn||ffi = 1 • □ 
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Given ip e Hq (T n U) let v v be the function defined in (3.24). The linear map 

tp e H^(T nU)^-2 [ (w+div r (i/'VrM + ) - ^div r (V>V r iO) dH N ~\ 
Jmu 

is continuous on Hq(TC] U) . If condition (4.3) is satisfied, then by Proposition 4.2 and by the 
Riesz Theorem there exists a unique element Tip e Hq (T n ?7) such that 



(2>,^ 



-2/ (v+div r (?/>Vrw + ) - Cdivr^Vrw"))^^- 1 (4.7) 

for every ^ e iJp (T n J7) . By this definition and (3.25) it turns out that 

ti 2 F((u,r);U)[p] = \\<p\\l-(T<p,<p)„ (4.8) 

for every <p £ Hq(T n E7) , provided (4.3) is satisfied. 
We now study some properties of the operator T . 

Proposition 4.3. Assume condition (4.3). Then the linear operator T : (Hq(T n U),~) — > 
(i?o(rn f/),~), defined by (4.7), is monotone, compact, and self-adjoint. 

Proof. By (3.25) and (3.26) we obtain 

(IV, p)„ = -2 / (v+div r (^V r u + ) - w-divr(^Vrw-)) dH^ -1 = 2 / |Vw v | 2 > 0, 
Jrnc/ J(7 



imu 

that is, T is monotone. 

Let ip n — ^ <p weakly in (if* (r fl f/), <~) . Then Proposition 4.2 implies that divr(<PnVrW ± ) 
converges to divr(<pVrw ± ) weakly in L 2 (rn U) . From (3.24) it follows that v Vn — 1 w v weakly 
in L ' (U\K;dU). By the compactness of the trace operator we have that (up to additive 
constants on the connected components of U\K whose boundary does not meet dU) converges 
to strongly in L 2 (T n U) . This is enough to deduce from (4.7) that T is weakly continuous, 
hence continuous. 

Taking <p = ip n and ip = Tp n in (4.7), we obtain that ||T^ n ||^ — > ||7V||~, which concludes 
the proof of the compactness of T . 
Using the Green identity 



/ (v+div r (^V r u+) - v-div r (ijV r u-)) dU 1 
Jmu 

= / (v+div r (pW r u+) - uTdivr^Vru"))^ 
Jmu 



iN-l 

'in 

/N-l 

uivr v r" ; — 'f^, uivr \<p v ru }) u, < 

imu 

it is easy to check that T is self-adjoint. □ 



Under the assumptions of Proposition 4.3 we can define 

Ai := max (T^,^ = ||T|U. (4.9) 
II v II — =1 

It is well known that Ai coincides with the first eigenvalue of T . The following proposition gives 
an equivalent characterization of Ai . 

Proposition 4.4. Assume condition (4.3) and consider the following auxiliary system in the 
unknown (y,<p) € Ll' 2 (U\K;dU)xH^(T n U) : 



A / Vv-Vzdx+ / [div r (^V r u + )z + - div r (ipV r u~)z~] dU 
Ju Jmu 



Vr^'Vr^H JV_1 + / aptpdH 1 *- 1 (4-10) 
imu Jmu 

+ 2 (div r (V>V r u + )v + - div r (^W r u-)v-)dH N - 1 = 
Jmu 

for all z e L ' (U\K;dU) and for all ip € H^Tf] U) . Then Ai coincides with the greatest A 
smc/i t/iai (4.10) admits a nontrivial solution (v,ip) 7^ (0,0). 
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Proof. It is enough to observe that under condition (4.3), A is an eigenvalue of T with eigenfunc- 
tion ip if and only if the pair (v v /X, p) (see (3.24) for the definition of v v ) is a nontrivial solution 
of (4.10). □ 

Remark 4.5. We note that the strong formulation of (4.10) corresponds to 

'Aw = inU\K, 

v = on dU, 

8^=0 onK\(TnU), (4.11) 

A9 ! ,u ± = div r (yVrM ± ) onTnU, 

-A r p + ap = 2V r w + • V r w + - 2V r u~ • V r «~ onTnU. 

Condition (4.1) can be characterized in terms of Ai , as explained in the following theorem. 

Theorem 4.6. Condition (4.1) is satisfied if and only if the following two properties hold: 

(i) {ip,<p)„ > for every p G H^(T H U)\{0} ; 

(ii) Ax < 1 . 

Proof. Assume that condition (4.1) is satisfied. Then by (3.26) we have 

-2 / (v+d„v+ -v-OvV^dH"- 1 = 2 f \Vv v \ 2 dx>0 
Jrnu Ju 

for every <p € Hq(TC\ U)\{0} , which implies condition (i). Once (i) is satisfied, condition (ii) is 
equivalent to (4.1) by (4.8) and (4.9). □ 

Upon assuming (4.3), we can also characterize the positive semidefiniteness of the second vari- 
ation (5 2 F((it, T); U) in terms of Ai . More precisely, we have the following. 

Theorem 4.7. Assume (4.3). Then condition (3.28) holds if and only if Ai < 1 . 

Proof. The fact that Ai < 1 implies (3.28) follows from (4.8) and (4.9), as in Theorem 4.6. 
Conversely, assume Ai > 1 and let p>\ be an eigenfunction of T associated with Ai . Then by 
(4.8) we have 6 2 F((u, T); U)[<pi] = (1 - Ai)||v?i||^ < 0. □ 

We conclude this subsection with a corollary, where we show that pointwise coercivity of the 
second variation 5 F((u, T); U) implies uniform coercivity. 

Corollary 4.8. Assume (4.1). Then there exists a constant C > such that 

S 2 F((u,T):UM>CM 2 Hl 

for every ip £ Hq (T fl U) . 

Proof. We first note that by (4.8) 

S 2 F((u,T); UM = \\p\\i (Tip, tp)^ > \\p\\i \\T\Up\\i = (1 - A^IMlL 
The conclusion follows from Theorem 4.6 and Proposition 4.2. □ 

Remark 4.9. If N = 2 condition (4.3) is always true. Indeed, by (3.22) the expression of a(x) 
reduces to H 2 (x). Therefore, by Theorem 4.6 condition (4.1) is satisfied in this case if and only 
if Ai < 1. In higher dimensions the situation is different. A counterexample can be constructed 
by considering as T r an unstable minimal hypcrsurface (i.e., a critical point of the area functional 
with nonpositive second variation) and then by choosing any function u defined in a tubular 
neighbourhood of T r , satisfying the first order conditions (3.2) and (3.22), and Vu~ = Vu + on 
T r . This can be easily done using Cauchy-Kowalevskaya theorem. The conclusion follows by 
observing that in this situation the bilinear form (4.2) reduces to the second variation of the area 
functional at T r . 
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4.2. A dual minimum problem. We introduce the linear operators 

A± : H^{T DU)^ L 2 (T n U), A ± <p := -2div r (^V r u ± ) 

and we denote by A± : L 2 {YC\ U) — ► i7 _1 (m U) the adjoint operators of A± with respect to 
the scalar product of L 2 (T n {/) ; i.e., for every i\> g L 2 (T n {/) and every 95 g i?g( r n u ) 

(A* ± tp,ip)=[ A±<pipdH N - 1 = -2 / div r (¥>Vru ± )VdH JV_1 , 
./mi/ ./rnt/ 

where (-, •) denotes the duality product in iJ _1 (T n {/) x Hq (T n {/) . We consider also the resol- 
vent operator R : i7 _1 (r n 17) — > (r n 17) , which maps any / g i7 _1 (r n ^) into tnc solution 
g i?o (r n £/) of the problem 

— Ap(/9 + aip = f in T n f7, 

v?e^(rni/). 

The operator R is well defined under the assumptions of Proposition 4.2. We note also that the 
operator T, introduced in (4.7), can be written as 

T<p = R(A* + v+-A*_v-) (4.12) 

for every ip e Hq(T n J7) , where v v is defined in (3.24). 
We introduce now the following dual minimum problem: 

min|2 J \\7v\ 2 dx: v g L^ 2 (U\K;dU), \\R(A* + v+ - A*tT)||„ = l}. (4.13) 

An argument similar to the one used in the proof of Proposition 4.3 shows that 

v 1 ^ R(A* + v+ - A*_v~) is compact from L^ 2 (U\K;dU) to H^TnU). (4.14) 

Exploiting this remark, it is not difficult to prove that the problem (4.13) admits a solution by 
the direct method of the Calculus of Variations. 

The following theorem, which is the main result of this subsection, provides a characterization 
of condition (4.1) in terms of the dual problem (4.13). 

Theorem 4.10. Assume condition (4.3). Then Ai = 1/fJ,, where \i is the value of (4.13). 
Moreover, condition (4.1) is satisfied if and only if (4.3) holds and fi > 1 . 

Proof. It is enough to prove that under (4.3) we have fi = 1 /Ai , as the second part of the statement 
will then follow by Theorem 4.6. 

Let (p g Ho(r n U) be such that \\ip\\^ = 1 and Tip = Xnp. Then by (4.12) we have 

R(A* + v+ - A*_v~) = Aip, (4.15) 

that is 

-AiA r ^ + \iatp = A* + v+ - A*_v~. 
Multiplying both sides by ip and integrating by parts, we obtain 

Ai / aip 2 dn N ~ 1 + Ai / IVrvfdW"- 1 = / «A+<p - w;A_^) rfW^" 1 . 
./rnt/ ./rnt/ ./rnt/ 

Using the fact that \\<fi\\~ — 1 and 2d u v^ = —A±ip, we deduce that 

Ai = -2 / («+cU+ - w-a„«-) rfW^" 1 = 2 / |Vtv| 2 da;, (4.16) 
./rnt/ Jt/ 

where the last equality follows from (3.26). By (4.15) the function v^/Xi is admissible for problem 

(4.13). Therefore, from (4.16) we infer that fi < 1/X\. 

To show the converse inequality, let v be a solution of (4.13). Then it is easy to see that there 

exists a Lagrange multiplier /x such that 

f Vv-Vz = f i (R(A* + v + - A*_v~),R(A* + z + - A*_z~))^ (4.17) 
Ju 

for every z g L ' ! (U\K;dU) . Choosing v as test function in (4.17), we deduce that 2^ = M- 



A SECOND ORDER MINIMALITY CONDITION FOR THE MUMFORD-SHAH FUNCTIONAL 



17 



We set (p := R(A* + v + - A*_v~) and ip := R(A* + z + - A*_z~). Then using the definition of ip 
and integrating by parts it turns out that 



/ atpiPdH N - 1 + [ Vr<p- VrV^ -1 = f (z+ A+tp - z~ A^) dH r 
Jmu Jmu Jmu 



yJV-l 

z si+yj — z /-L—{fj)ur 

imu Jmu Jmu 

in other words 

{R{A* + v+ - A*_v-),R(A* + z+ - A*_z~))^ = (^,V)~ = / {z + A +V - z~ A_ip) alU"' 1 . (4.18) 

Jmu 

From (4.17) and (4.18) it follows that j^v satisfies (3.24), which implies that v = [iv v . Therefore, 
by (4.12) we have that 

Tip = -R(A* + v+ - A*_v~) = -ip; 
fj, [i 

i.e., l/jj, is an eigenvalue of T. This implies that < Ai and concludes the proof of the 

theorem. □ 

In the next corollary the dependence of Ai and /j, on the domain will be made explicit. In 
particular we will show that they depend monotonically on U . 

Corollary 4.11. Assume that condition (4.3) is satisfied. Let Ui,U 2 C Q be admissible subdo- 
mains for (u,K) such that U\ C U2 and TDUi = T(~]U2- Then Xi(Ui) < \\{U2) ■ In particular, 
if condition (4.1) is satisfied in U2 , then it also holds in U\ . 

Proof. As dU\ a 8l{Ui\K) , we have that if v e L^ 2 (U\\K; dU\) , then the function v given by 
v — v on Ui\K and v = on U 2 \Ui belongs to L L ' 2 (U 2 \K; dU 2 ) . Therefore, if v is an admissible 
function for the problem (4.13) in U\, then v is an admissible function for the problem (4.13) in 
L>2- Hence ^(Ui) > ^(^2)- The conclusions follows from Theorems 4.10 and 4.6. □ 

The following corollary will be used in the next section (see Remark 5.2). It shows that Ai and 
H are continuous along decreasing sequences of open sets. 

Corollary 4.12. Assume that condition (4.3) is satisfied. Let U n C £1 be a decreasing sequence 
of admissible subdomains for (u, K) . Assume also that the open set U defined as the interior part 
of r\^ =l U n is an admissible subdomain for (u 7 K) and that T n U n = T n U for every n. Then 
Ai (£/„)-> Xi(U). 

Proof. In view of Corollary 4.11 it is enough to show that lim n Xi(U n ) < Ai(C7) . By Theorem 4.10 
this is equivalent to prove that 

Umn{U n )>n(U). (4.19) 

n 

Let v n be a solution of (4.13) with U replaced by U n . Then the function v n given by v n — v n 
on U n and v n = on U\\U n belongs to L^ 2 (Ui\K;dUi) and 2 f v |Vw„| 2 dx — n(U n ) < n(U) . 
Hence there exists a subsequence (not relabelled) v n and a function v G L ' (Ui\K;dUi) such 
that v n ~ ^ v weakly in Lq' 2 (Ui\K; dU\) . Clearly v — a.e. in U±\U , which in turn implies that 
the restriction v of v to the set U belongs to L ' (U\K; dU) . Recalling also (4.14) we infer that 
v is admissible for problem (4.13) and thus 

\\m.li(U n ) =lim2 / \Vi n \ 2 dx>2 | Vv\ 2 dx = 2 / |Vw| 2 dx > n(U), 

n n JUi JUi Ju 

which shows (4.19) and concludes the proof. □ 

5. A SECOND ORDER SUFFICIENT MINIMALITY CONDITION 

In this section we show that any critical point satisfying the second order condition (4.1) is a 
local minimizer with respect to variations of class C 2 of the regular part T r of the discontinuity 
set. Critical points which are C 2 -local minimizcrs (in the sense of Definition 3.14) play in our 
context the same role of weak minimizers in the classical Calculus of Variations, as made precise 
by the following theorem. 
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Theorem 5.1. Let N < 3 and let 0, (u,K), U, and T be as in Definition 3.3. Assume in 
addition that (u,K) is a critical point in U with respect to T and that (4.1) is satisfied. Then 
(it, K) is an isolated C 2 -local minimizer in U with respect to V . 

Remark 5.2. We observe that in the statement of the theorem we can assume without loss of 
generality that T CC U . Indeed, if this is not the case, setting V := m U , we can find an 
admissible subdomain U' D U such that V CC U' and S 2 F((u, V); U') is positive definite on 
i?Q (r') . The existence of such a domain U' is guaranteed by Corollary 4.12. It is now sufficient 
to show that (u,K) is an isolated C 2 -local minimizer in V with respect to T' , since this implies 
in particular the thesis of Theorem 5.1. 

In view of the previous remark we may assume in the remaining part of the section that 

r cc u. 

In order to prove Theorem 5.1 we need some auxiliary results, which are contained in the next 
lemmas. For every S > we define the 5 -neighbourhood (A)s of an arbitrary set A C R N as 

(A)s := {x £ R N : dist (x, A) < S}. (5.1) 

For notational convenience we set 

V s := {$ £ C 2 (U;U) : $ diffcomorphism, $ = /on (Jfn U)\T, < \\§ - I\\c < 6} 

for every 6 > 0. We fix So > such that the orthogonal projection Ilr,. on T r is well defined 
(and smooth) in (T)s n U and for every $ £ T>s there exists a unique ip £ Cq(T) such that 

= $(r) = {x + <p(x)v(x) ■. x e r}. 

We can then define in (T)g n U the vector field 

:= (ipv) o n Tr (5.2) 
for every $ £ T>g . Moreover, we consider the bilinear form 

N-l , / Y7_ „Q Y7_ „/, 



(i?,^)~,*:=/ a^dH N - 1 + V r *tf •V r<t Vd?T'- 1 (5.3) 

for every ip £ Hq(T$), where 

a$ := 2B*[Vr.uJ,Vr.uJ] - 2B«[Vr.«i , Vr.u^] - l B *| 2 

(here and in the sequel we use the same notation as in the previous sections). 

In the next lemma we prove that the H 1 -norm on T$ can be controlled in terms of the norm 
|| • ||^,$, uniformly with respect to $. 

Lemma 5.3. There exist C\ > and 5i £ (0, S ) such that for every $ £ Vg 1 we have 

IW^(r.)<Ci|W~,* (5.4) 

for every -0 £ Hq (r$ ) . 

Proof. As (u,K) satisfies the second order condition (4.1), by Theorem 4.6 and Proposition 4.2 
we have that there exists a constant C > such that 

U\\m (r) < CU\\i (5.5) 

for every tb £ Hq(T). Setting A'/ := sup$ eI7 sup xernU Jq>(x) (\(Dr$)~ T (x)\ 2 + 1), by the area 
formula (2.7) we obtain 

= jf (|V o $| 2 + |(£) r $)- T [V r (V o *)]| 2 ) J$ d?^" 1 

< My (|V o$| 2 + |V r (^o $)| 2 )dW Ar ~ 1 <MC\\tpo (5.6) 
where in the last inequality we used (5.5). 
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Let e be a positive constant that will be chosen later. By classical elliptic estimates (see, 
e.g., [16, Theorem 3.17]) we have that is C l a up to T$ for some a G (0, 1), with C 1,Q -norm 
uniformly bounded with respect to $ G T>g . It follows that the map $ (a$o<f>) J $ is continuous 
from T>s , endowed with the C 2 topology, into L°°(T) . In particular, there esists Si G (0, <5o) such 
that || (a$ o $) J$ — a|| L oo(p) < e for every $ G , and, taking o\ smaller, if needed, we can also 
guarantee that sup$ eX>4i sup xerni7 J $ 1 (x) \(D r $) T (x)\ 2 < l+e and sup$ eI7{i sup xern[/ J^a;) < 
1 + e. Hence, using also the area formula (2.7), we have 

< y'( a$ o$)|v;o$| 2 j$dw A, - 1 + J\v T {i'o^)\ 2 dH N - 1 +e J\i,o<^\ 2 dn N - 1 

< f a^dH"- 1 + {l + e) f \V T ^\ 2 dH N - 1 +s{l + e) f |V| 
•^r* •/r* •/r* 



r 

2 dn N - 1 



2 i , _\ / /l„,.|2 , |V7_ „/.|2\ juN-1 



< IC,* + £(l + £) / (M' + |Vr.Vr)dW 

Choosing e > such that MCe{\ + e) = \ , the thesis follows from (5.6) and the previous 
inequality with C\ := \J2MC . □ 

From the previous lemma, Proposition 4.2, and Remark 4.1 it follows that for every $ G X?^ the 
bilinear form (•, -)~,* is a scalar product on Hq(T^>) , so that, similarly to (4.7), we can introduce 
the operator T$ : ifg(r$) — > i/p(r$) defined by 

(T*i?,V)~,* = -2 / (^^divr.^Vr.O-^^divr.^Vr.ui))^- 1 , (5.7) 
Jr* 

where € fJ' 2 ([/\FT$; <9£7) is the solution of 

/ Vv*, 9 -Vzdx+ [ {div T *(Wr*u+)z + -div r *(Wr*Us)z-)dH N - 1 =0 
JV Jr* 

for all z G LQ' 2 (U\K^;dU) . By Proposition 4.3 and Remark 4.1 the operator T$ is monotone, 
compact, and self-adjoint for every <!> G 2?^ . Moreover, we have the following property. 

Lemma 5.4. Assume N < 3. For $ G T>s 1 let Ai,$ denote the norm of T$ on ffo(r$) endowed 
with the norm || • and let Ai := Aij. ITien 

limsup Ai $ < Ai. (5.8) 

||*-/|| 2-0 

Remark 5.5. It is actually possible to prove that Ai,$ converges to Ai , as ||$ — I\\c 2 — » 0, but 
this is not needed in the sequel. 

Proof of Lemma 5.4- Assume by contradiction that (5.8) fails. Then there exist Aoc > Ai , <&„ — > I 
in C 2 -norm, ip n G C^°(r$ n ) with ||<Pn||~,$„ = 1, and w n G L ' (U\K$ n ;dU) solution to 



/ Vw n -Vzdx + / (div r$?i (93„Vr 4 . n M$ n )2; + - div r<i , n (<£ n V r<i . n «<i> n )-2 ) cW^ : =0 
JV Jr*,, 

for all z G Lq 2 (U\K& n ; dll) , such that 

(7* n </j n , <p„)~,* n = 2 / |Vw„| 2 dx — > Aoc > Ai. (5.9) 
Let ?D„ := w n o $„. Then w n G F '' ! (U\K;dU) satisfies 

y A„[Vw)„,Vz] dx + y(divr 4?i (^ l Vr t ,„<J)o$„J <I , 7i z + dH Ar - 1 

- jT (divr. n (^ n Vr. n «i B )) ° $nJ*„z - rfH*- 1 = 
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for all z e Lq 2 (U\ K; dU) , where A n := ^"j^f " ° $ n with := , while J$ n is the 
(N — 1) -dimensional Jacobian of <j>„ . Moreover, it is easily seen that 

lim 2 / \Vw n \ 2 dx = lim 2 / | Vw„| 2 = Aoo. (5.10) 
n Ju n Ju 

We finally set y>„ := c n (fi n $n , where 

c„ := ||<p n o $„||- 1 1, (5.11) 

and we consider the function defined by (3.24) with ip replaced by (p n . To conclude the proof 
of the lemma it will be enough to show that 

lim/ \V(v<2 -w n )\ 2 dx = 0. (5.12) 
" Ju 

Indeed, by (5.9) and (5.10) this would imply 

Ai > lim (Tif n , <p n )~ = lim 2 / |Vt><2 | 2 <ix = lim2 / | 'Vz()„ | 2 dx = > Ai, 
n n Ju " Ju 

which gives a contradiction. 

In order to prove (5.12) we observe that z n :— vq, n — w n solves the problem 

z n e Ll' 2 (U\K;dU), 

[ A n [Vz n ,Vz]dx- [ (A n - I)[Vv 0n ,Vz}dx+ [ (h+z+ - h~z~) dH N ~ l = 
Ju Ju Jr 

for all z € Ll' 2 (U\K;dU), where ft± := div r (^„V r w ± ) - ((div r<i . n (^„Vr in u| n )) o $„) J$„ . Since 

A n — 1^0 in C 1 -norm and vq, n is bounded in LQ' 2 (U\K;dU) , we have that (A„ — -0[Vv^J 

converges to strongly in L 2 (U;R N ) . Hence (5.12) follows once we show that — > in 

H-i(T). 

To this aim let ( e C c °° (T) . Then we have 

f ((divr.J^Vr^utjJo^J^CdH"-^ f div r<tn (^V r#n 4 n )(C ° *») dH 1 
Jr Jr<s,„ 

Jr^ n 

= - I ^„(^r$„)" T o*„[V r (4 o<$> n )oy n ]-(D r ^ n ) T [(V r ()oy n ]dH N -' 

Jr *n 

= - J c i ; 1 ^( J Dr$„)- 1 pr$„)- T [V r « o $„), V r C] J*„ ■ 

= ^c^Mivr^nJ^J^r^)- 1 ^^)- 7 ^^ o* n )])CdW 7 
where we repeatedly used the area formula (2.7). It follows that 

ht = div r (^„Vru ± - W*„(A-*n) _1 (I>r* n r r [Vr(ui B o *„)]). (5.13) 
We claim that for every a e (0, 1) 

V r (ug n o $„) -» Vr^ in C°' a (r ; R w ). (5.14) 
To prove this we observe that y„ := o $„ — u solves 

y„ e L^ 2 (U\K;dU), 

/ A„[Vy n , Vz]dx+ / (A n -i")[Vu,Vz]cZx = 

for all z G L 1 / (U \ K ; dU) . As A„ -» 7 in C 1 (17; R ArxAr ) , we deduce by standard elliptic 
estimates (see, e.g., [16, Theorem 3.17]) that y n — > in W 2 ' P (V\T) for every p and for a suitable 
neighbourhood V of T. This provides (5.14). 
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It is now convenient to set ip^ := c^ 1 J< s>n (D r $ n )~ 1 (D r $ n )~ T [S/r(u& n ° — Vr^. As the 
matrix 4jflr$ n ) _1 (Dr3 ) n) _T converge to I in C 1 (r; K 7VxA ') , claim (5.14) and the convergence 
in (5.11) imply that for every a € (0, 1) 

V£->0 in C°' a (T;R N ). (5.15) 

Let us fix a e (0, 1) and p > 1 such that (2a — l)p > 2iV — 2. As <^„ is bounded in Hq(T) and 
A < 3, by the Sobolev imbedding theorem (p n is bounded in L P (T), too. Adding and subtracting 
the term (p n (x)^(y) and using the Holder continuity of we can estimate the Gagliardo 
i?2 -seminorm of (fn^n as follows: 

JJr dH {X)dH {V) 

< 2||tf±||£ oc(r) ||$> n |& va(r) + 2Ut\\ 2 C o, a{r) JJ^ \$n{x)\ 2 \x - y\ 2a - N dH N -\x) dH N -\y) 

< 2||^|| 2 c o, Q(r) (||^|| 2 ffl/2(r) 

+ H N -\T)l ||^||| P(r) ( jf jf |.x - y\^-^ dH N -\x) dH N -\ y ))^). 

By our choice of a and p the last integral in the previous formula is finite. Thus, using the 
boundedness of (p n in H^(Y) and in L P (T), we deduce from (5.15) that 

<p n tf^0 mHi(T;R N ), 

which in turn gives — > in if _ i(r) by the definition of ^ ■ □ 

Remark 5.6. The assumption N < 3 in Lemma 5.4 can be removed if we require $ to converge 
to I in the C 2 ' Q -norm for some a € (0,1). Indeed, arguing by contradiction as before, the 
proof reduces to show that — > in if~3(r). Since <!>„ converge now to I with respect to 
the C 2 ' Q -norm, we deduce by standard elliptic estimates that y n — > in C 2 ' Q -norm up to L, 
so that V r (u|„ o $„) -» V r u ± in C 1,Q (r; R w ) . As J$„(D r *n) _1 ( £) r$ n )~ T converge to 7 in 
C 1 ' a -norm, we "have that ip± -> in C^ Q (L; K w ) , hence £„Vn -> in ^(T; R w ) , which implies 
h± -» in ff-3(r). 

Proof of Theorem 5.1. First of all, we note that it is enough to show that there exist 8 € (0, <$i) 
and c> such that for every $e%nC°° (U; 77) , with supp ($ - 7) n T CC T and T$ ^ V n 17, 

F((u, 7C); [/) < F((««, 17) - c||X* • Ml W*)' (5.16) 

where, we recall, = $(A"n J7) and A"$ is defined in (5.2). Indeed, the statement would then 
follow by approximating in the C 2 -norm any $ € T>$ with diffcormophisms having the properties 
above. 

The strategy will be the following. Given <E> G V$ n C°°(U; U) with (5 < <5i , we consider an 
admissible flow $ t for T in f7 which coincides with I + tXg, in the ^-neighbourhood of L. Setting 
:= F((u<s> t ,K<£ t ); U) , we shall show that there exist 8 < 8\ and c > such that 

g'i{t) > 2c\\X<i, ■ v*||^i (r#) for every f e [0, 1] and every $ e V s n C°°(C7;T7). (5.17) 

As s4(0) = 0, condition (5.17) will then imply 

F{{u,K);U) = s*(0) = (l-t)«^(t)dt 

< ^(l)-2c||X <E> .^|| 2 ?1(rs) ^ (l-t)dt 
= F((u*,iif$);{7)-c||X*-i/*||| r i (r4) , 



that is (5.16). 
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Let us prove (5.17). Using (5.3), (5.7), and the fact that Xj t = X$ , so that Z^ t — 0, we have 
by Remark 3.10 that for every t £ [0, 1] 

g'i{t) = - (T$ t (X$ t • v^ t ),X^ t ■ y^ t )^^ t + \\X^ t ■ ^*JU i$t 

M-2Xl t • V r . t (X* t -i/*J +B $t [4 t ,4j +ff* t (X* t -^J 2 )^- 1 , (5.18) 



X 4 



where we recall that X$ stands for (I — v$ t <g> i/$ t )A$ t and f t — | Vr # u$ t | 2 — I Vr # u$ t | 2 + • 
As (u,K) satisfies the second order condition (4.1), it follows from Theorem 4.6 that Ai = 
Xij < 1 . Hence by Lemma 5.4 there exists S 2 £ (0, <5i) such that 

Ai,*<|(Ai + l)<l (5.19) 
for every $ £ T>s 2 . By taking 62 smaller, if needed, we can also guarantee that 

|||^*-^*||iji(r # ) < l|A$ t ll/ *tlljTi(r* t ) ^ 2 II^* ' v^Wh 1 ^) (5.20) 
for every <E> £ T>s 2 and every t £ [0, 1] . Using the definition of Ai^ and invoking (5.4), we deduce 
-(T$ t (A$ t • z/$ t ),X$ t • i/^J^,^ + \\X<s> t ■ ^* t |U i$t > (1 - Ai^JHX^ • !A£ t |U ;< j, t 
> ^^(l-AOHX^.^JI^^) > ^(l-AOIll^^H^^), (5.21) 

where the last two inequalities follow from (5.19) and (5.20). 

Choosing 5 2 smaller, if needed, we also have that v§ — voTi Tr + p$> with Hp^Hc 1 < \ f° r every 
$ £ Vg 2 . As \X&\ = |X$ • (y o IIr r )| , we deduce that 

I A* I ^ |A* • I + I A* • (p$ rir r ) I < I A$ • I + ^ I A$ I, 

hence 

\X*\<2\X*-v*\ (5.22) 
for every $ e . Moreover, as the C 1,Q -norm of u% on T$ is uniformly bounded with respect 
to $ £ T>s 2 , one can show that the map 

$ e V S . 2 1 ^ |||Vr 4 u $ | 2 - |V r$ 4| 2 + ff*|U~(r*) 

is continuous. In particular, as it vanishes at $ = /, for every s > there esists <5 e (0,(5 2 ) such 
that 

|||Vr t u $ | 2 - |Vr*4l 2 + #*IU~(R*) <£ 
for every Hence, there exists a constant c > such that for every t £ [0, 1] 

f ft(~2Xl t ■ V r . t (A $t ■ + B $t [A| ( , A|j + H* t (X* t ■ v* t ) 2 ) dW 7 

> - 2e ||-^* t ||i,2(r #t )||Vr #t (-y* t • ^* t )||L 2 (r 4t ) — c e ||^* t Ili2( ra>t) 

> -4(l + c)e||X* t -i/* t ||^ 1(r4t) > -8(l + co)£||AV^||^ 1(r$) , 

where the last two inequalities follow from (5.20), (5.22), and the fact that A$ t = jX$ t . Choosing 
e so small that 8C 2 (1 + co)e < g (1 — Ai) , claim (5.17) follows from the previous inequality, (5.21), 
and (5.18), with c jLC"f 2 (l - Ai). □ 

Remark 5.7. We observe that in the course of the proof of Theorem 5.1 we made use of the 
technical assumption N < 3 only in Lemma 5.4. Thus, by Remark 5.6 the following weaker 
version of Theorem 5.1 holds in dimension N > 3. If (u, K) is a critical point in U with respect 
to r satisfying (4.1), then for every a £ (0, 1) there exists 8 > such that 

f \Vu\ 2 dx + H N - r {KnU) < f |Vw| 2 dx + W Ar - 1 ($(AnC/)) 
Ju Ju 

for every C 2 ' a -diffeomorphism $ on U with $ = I on (KnU)\T, L$ ^ Tf\U, and ||$-J|| C 2, a < 
S, and every v £ L 1,2 (JJ\<t{K n U)) with v = u H N ~ 1 -a,.e. on dU . In other words, (u, K) is an 
isolated C 2 a -local minimizer in U with respect to L for any a £ (0, 1) . 



jN-l 
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6. Stability and instability results 

We start with two results of stability in small domains. In the first proposition we show that 
(u, K) is an isolated C 2 -local minimizer in a tubular neighbourhood (r% of T (see (5.1) for the 
definition of (r%), provided condition (4.3) is satisfied. 

Proposition 6.1. Let ft, (u,K), and T be as in Definition 3.3. Assume that (u,K) is a 
critical point in O with respect to T and that (<p,<p)„ > for every ip £ i7g(r)\{0}. Assume 
furthermore that (r% is an admissible subdomain for (u,K) (in the sense of Definition 3.1) for 
every e £ (0,£o)- Then there exists E\ £ (0, eo) such that for every e < E\ the second variation is 
positive in (T) £ ; i.e., 

5 2 F((«,r);(T) e )M>0 

for every ip £ Hq(T)\{0} . In particular, if N < 3, (u,K) is an isolated C 2 -local minimizer in 
(r% with respect to T, while if N > 3, (u,K) is an isolated C 2 ' a -local minimizer in (T) E with 
respect to T for any a £ (0, 1) . 

Remark 6.2. If N = 2, by the previous proposition and Remark 4.9 it follows that every critical 
point is an isolated C 2 -local minimizer in a tubular neighbourhood of a compact subarc of the 
regular part of the discontinuity set. This is in agreement with the result in [11], where in fact a 
stronger minimality property is proved. Instead if N > 3 , there exist critical points whose second 
variation is nonpositive in every tubular neighbourhood of the regular part of the discontinuity 
set. This follows from Remark 4.9, where it is shown that condition (4.3) may fail. 

Proof of Proposition 6.1. By Theorem 4.10 it is enough to show that 

lim M( r )e) = +oo, (6.1) 

where /j,((T) e ) is the value of (4.13) with U replaced by (1% . Assume by contradiction that 
(6.1) fails. Then there exist C > 0, s n -> 0+ , and v n £ Ll' 2 ((T) £n \K;d(T) £ J such that 
\\R(A* + v+-A*_v-)\\^ = l and 

/ \\7v n \ 2 dx<C. 

By setting v n = on £1 \ (r% n we have that v n is a bounded sequence in L^ 2 {VL \ K; dfl) . Since 
the measure of (r) £n goes to zero, we deduce that v n converge to weakly in L ' 2 (£l\K; 90) . 
As the operator (4.14) is compact, we conclude that R(A* + v+ — A*_v~) converge to strongly in 
H^(T), which contradicts \\R{A* + v+ - A*_v~)\\~ = 1. 

The last part of the statement follows from Theorem 5.1 and Remark 5.7. □ 

In the next proposition we prove that the generic critical point (u, K) is stable with respect to 
C 2 perturbations with small support. 

Proposition 6.3. Let fl, (u,K), U , and T be as in Definition 3.3 and assume in addition that 
(u, K) is a critical point in U with respect to T . Then there exists R > such that 

S 2 F((u,T);UM>0 (6.2) 

for every ip £ H^(T C\ U)\{0} with diam (supp ip) < R. 

Remark 6.4. Arguing as in the proof of Theorem 5.1, one can show that the thesis of Proposi- 
tion 6.3 implies the following minimality property in dimension N < 3: there exists 5 > such 
that 

f ivufdx + n"- 1 ^ nu) < f |v«| 2 dx + w Ar - 1 ($(if nu)) 

Ju Ju 
for every $ £ V s with diam (supp (<& - I) n T) < R and r$ ^ T n U , and every v £ L 1 < 2 (U\ 
&(KnU)) with v = u W^-i-a.e. on dU . 
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Proof of Proposition 6.3. As an easy consequence of Poincare inequality, we infer that there exists 
Ro > such that for every isf, with V nil CC V CC T r , we have 

{<p,<p)~>0 (6.3) 

for every (p e Hq(T n E/)\{0} with supp C B Ra (x) . By Proposition 4.2 the bilinear form (4.2) 
defines an equivalent scalar product on the subspace 

H Xtr := {<yd € Hq(T n £7) : supp ip C B r (a;)} 

for every r < Rq and every lef. Thus we can define by duality the operator T x , r : H x , r — > i? x>T . 
satisfying 

(Tx^rf, = -2 / «div r (^V rU + ) - u-divr^Vru")) cW^" 1 (6.4) 

for every ip,ip G H x ^ r . The operator T XiT . may be thought of as a "localization" of T and 
turns out to be compact and self-adjoint. We note that by the representation formula (6.4), if 
B ri (a;i) C B r2 (x2) , then for every ip <G H Xliri the function T Xliri ip coincides with the orthogonal 
projection (with respect to (•, -)^) of T X2 ^ r2 tp on H Xi ri . Moreover, using (6.3) and arguing as in 
the proof of Theorem 4.6, one can see that inequality (6.2) is satisfied for every ip e H x , r if and 
only if 

max (T x r ip, tp)r* < 1. 

II Vll ~=1 

Therefore, to conclude the proof it is enough to show that 

lim max (T x , r ip, <p)^ = uniformly with respect to x £ T n U. (6.5) 

IMI~=1 

Assume by contradiction that (6.5) fails. Then there exist C > 0, x n e m U, r n — > + , and 
<Ai € H Xnt r n such that ||<£ n ||~ = 1 an d 

( T 'x n ,r„Vn,¥'n)~ > C- (6-6) 



Without loss of generality we can assume that x n — > x e r n £7 and B rn (x n ) C Br q (x) for n 
large enough. In particular, by the projection property mentioned before this implies 

{ T x n ,r n ¥>n,<Pn)~ = {T x ,R <fi n , ¥>n)~- (6-7) 

As ||Vn||~ = 1 an d the measure of the support of ^„ tends to zero, we conclude that <p n — ^ 
weakly in i?o (r n £7) . Since T x ,r is compact, it follows that T Xt R (p n — > strongly in Hq (V n £7) 
and in turn, (T x ^ ip n ,ip n )^ — > 0. By (6.7) this contradicts (6.6). □ 

We conclude this section with an example of instability in large domains. A related explicit 
example will be discussed in the next section. Let w : l^ 1 — > M be an affine function. We 
consider as critical point the pair (u, K), where for every x = (x',Xn) € R N 



w(x') for xn > 0, 
-w(x') for xn < 0, 



u(x) := 
and K = Y r = {xn = 0} . 

Proposition 6.5. There exists i? > such that the second variation S 2 F((u, KC\Br{x)); Br{x)) 
is nonpositive for every x <G K and every R > Rq . 

Proof. We first note that, as a = in this case, condition (4.3) is satisfied. Therefore, we can 
consider the operator T defined in (4.7). By Proposition 4.4 there exists a nontrivial solution 
(v,(p) e J Lj' 2 (Bi(0)\if;a J Bi(0))xiJ 1 (ifn Bi(0)) of (4.10) with U = Bi(0) and A = Ai(Bi(0)). 
For every xq € K and every r > let us consider the functions v r e L ' (B r (xo)\K; dB r (xo)) 
a,nd (p r e Hq(K r\B r (x )) defined by v r (x) := v{^=^) and <p r (x') — rip( x ~ x ° ). It is easy to see 
that (v ri ip r ) is a nontrivial solution of (4.10) with U — B r (xo) and A = rAi(7>i(0)) . Therefore, 
by Proposition 4.4 we have Xi(B r (xo)) > rAi(Bi(0)). The conclusion follows by Theorem 4.7 
choosing R = l/Ai(Bi(0)). □ 
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7. AN EXPLICIT EXAMPLE 

As a final application of the results of the previous sections, we discuss an explicit example, for 
simplicity in dimension 2 . In O = R 2 we consider the function 



u(x,y) 



x for y > 0, 
—x for y < 0, 



whose discontinuity set is given by K = T r = Mx{0}. For every Lipschitz bounded domain U in 
R 2 we recall that \\{U) denotes the constant introduced in (4.9), corresponding to this choice of 
u, K , U , and to T = T r (lU . We will compute explicitly the value of Ai for rectangles which are 
symmetric with respect to K . 



Proposition 7.1. Let U = (x , x + l)x(—y , y ) , y > 0, and let T = (xq, x + £)x{0} . Then 



X 1 (U) = ^ tanh 2 ^, (7.1) 



so that the second variation is positive if and only if 

M t anh^a<l. (7.2) 

In particular, if (7.2) holds, then (u, K) is an isolated C 2 -local minimizer in U with respect to 
T ; if ^tanh^^ > \, then (u,K) is not a minimizer in U . 

Remark 7.2. In [11] it is proved that, if a condition stronger than (7.2) is satisfied, then a 
stronger minimality property holds. More precisely, from the results of [11, Section 4] it follows 
that there exists a constant c < 1 such that if ^ tanh ^ < c , then (u, K) minimizes F among 
all competitors in SBV(Q), whose extended graph is contained in a sufficiently small tubular 
neighbourhood of the extended graph of u . 

Proof of Proposition 7.1. We choose v{x, 0) := (0,1) as an orientation for T. Let (v,<p) be a 
nontrivial solution of (4.11). By symmetry we have v(x,y) — v(x,—y); thus, setting R := 
(xq, x + £)x(0, y ) , we have that (v, ip) solves the problem 




Combining together the two conditions on T , we deduce that 

\d y v = — 4(u — c) on T, 

where c := j f* 0+l v(x, 0) dx. The computation of Ai amounts to the identification of the largest 
A such that there exists a nontrivial solution of 

Av = in R, 

v = on dR\T, (7.3) 

A d y v = — A(v — c) on T. 

Expanding v in series of sines and taking into account the first two conditions of the system, we 
have that 

v(x,y) = ^c„sin(^ (x - x )) sinh(^(y - y)) 

n 

with c n e M. Differentiating with respect to y and imposing that d y v has zero average on T, we 
obtain the condition 

c„cosh(^) = 0. (7.4) 

n odd 



26 F. CAGNETTI, M.G. MORA, AND M. MORINI 

Expanding also c in series of sines on [x , x a + £], one can see that the last condition in (7.3) is 
equivalent to 

A 7 S UCn ^H^f") sin(^f(x - x )) 



= 4^c„sinh(^)sin(^(: C -xo))-16c ]T ^ sin(^(:c - x )), 

which implies 



nGN neN 

n odd 



Af nc n cosh 2™> = 4c„ sinh 2™> for n even, (7.5) 
Af nc„ cosh ^ = 4c„ sinh ^ - 16c^ for n odd. (7.6) 
From (7.5) we deduce that either c„ = for every n even or there exists an even number n such 



that 



A = 4-Ltanh^. 



Clearly the biggest A which falls in the latter case, corresponds to n — 2 and hence, 

Ai(*7)> f tanh^a. (7.7) 
If c n = for every n even, it follows from (7.4) and (7.6) that c ^ . Hence (7.6) is equivalent 



to 

16 



" n7T 4 sinh S^ls.-xif cosh ^ 
for every n odd. Condition (7.4) and the fact that c ^ finally yield 

S ^2 4 tanh nzn«_^V = - ( 7 ' 8 ) 

neN n « « 

n odd 

By (7.7) the proof is concluded if we show that the previous equation has no solution in the 
interval (^ tanh +oo) . If A > ^ tanh ^f- , all the terms of the series in (7.8) are negative 
(since tanhx/x is decreasing for x > 0), so that we can restrict our attention to the interval 
[f tanh tanh ^f). Let g(X) be the function given by the left-hand side of (7.8). It is 

easy to see that g is monotone increasing in tanh ^j 2 -, tanh ^r 1 ) . Hence it will be enough 
to prove that tanh -^) > 0. This is equivalent to 



1 >V^ I (79) 



4 tanh 2EJ(a _ 2 tanh ^f- ^ n 2 2 tanh - 1 tanh 

n odd 



Using the inequality 
for every n > 3 and the identity 



2tanh 2 ^ - -tanh^a > 2 tanh ^ - - tanh ^ 

1 n - - 3 



„2 ~ Q 



n 

n>3 
n odd 



inequality (7.9) will be proved if we show 

i tanh _ i tanh 3im ^2 

^ * =-3 5—^- > 1. 7.10 

tanh If- - \ tanh ^ 8 V ; 

Applying the addition formula for the hyperbolic tangent it is easy to see that 

i tanh(2x) — | tanh(3x) 5 — tanh 2 x 

tanh x - \ tanh(2x) 3(1 + 3 tanh 2 x) 

for every x > 0. By this identity it is then clear that the left-hand side of (7.10) is a decreasing 
function of yo and its infimum is equal to i > ^ — 1. This concludes the proof of (7.9), and in 
turn of (7.1). 



A SECOND ORDER MINIMALITY CONDITION FOR THE MUMFORD-SHAH FUNCTIONAL 



27 



The last part of the statement follows now from Theorem 4.6, Remark 4.9, Theorem 5.1, and 
Theorem 4.7. □ 

8. Appendix 

In this section we collect some auxiliary results, which are needed in the proof of Theorem 3.6. 
We start with a proposition where the regularity properties of the map 1 1— ► u<s> t are investigated 
(see (3.3) for the definition of u$ t ). We give only a sketch of the proof. 

Proposition 8.1. Under the assumptions of Theorem 3.6, let u t := u$ t o $ t and v t :— u t — u . 
The following properties hold: 

(i) the map t ^ v t belongs to C°°((-l, 1); L^ 2 {U\K- dU)) ; 

(ii) for every x G T let B be a ball centered at x such that B C U , B r\T s = , and B\T 
has two connected components, B + and B- . For every t G (—1,1) let uf denote the 
restriction of u t to B±. Then the map u ± (t, x) := uf(x) belongs to C°°((— 1,1) x B±). 

Proof (Sketch). In order to prove part (i), it is enough to show that for every to G (—1, 1) the 
map t i — ► Vt is smooth in a neighbourhood (to — e, to + e) ■ For simplicity we consider only the 
case to = (the general case can be treated similarly). 
First of all, wc note that by (3.3) the function v t solves 

/ A t [Vv t , \7z]dx+ I A t [Vu, \7z]dx = for every z G L ~' 2 (U\K;dU), (8.1) 
Ju Ju 

where A t := o $ t and * t := ^ . 

Let us consider the map T : (-1, 1)xLq' z (U\K; dU) -» L^{U\K; dU) defined in the following 
way: for every t G (—1,1) and every v G L^' 2 (U \ K '; dU) the function T{t,v) is the unique 
solution C G L^ 2 {U\K-dU) of 

f VC-Vz= [ A t [\7v,\7z]dx+ [ A t [\7u,\7z]dx for every z G L^ 2 {U\K;dU). 
Ju Ju Ju 

It can be checked that T is of class C°° , JF(0, 0) = (as (u, K (~l U) G ^4 res (L r ) by assumption), 

and ^^"(OjO) is an invertible bounded linear operator from LQ' 2 (U\K;dU) onto itself. Hence, 

since v t satisfies !F{t, v t ) — by (8.1), part (i) of the statement follows from the Implicit Function 

Theorem. 

Let us fix x G T and let B, B + , and B- be as in part (ii) of the statement. Let v' to be the 
derivative of t i— > v t with respect to the L 1,2 -norm, evaluated at some to, which exists by part 
(i). We claim that 

v' to =v to in B and G C 1 ((— 1, l)xB±). (8.2) 

To this aim we first observe that by (8.1) the function w h := j^(v to+h — v to ) is the solution of 

/ A t0+h [\7w h ,\7z]dx+ [ \{A t0+h - A to )[\7u to ,\7z]dx = for every z G L^ 2 (U \ K; dU). 
Ju Ju 

(8.3) 

By standard elliptic estimates for every p > 1 the restrictions to B± satisfy 

\\ w h\\w 2 .P(B ± ) < Cp 

for some constant C p independent of h. We deduce that v^ g+h — > v^ o and — ► (w^)* in 
C 1 (i3±), as h — » 0. In particular, 

(t, x) i — ► Vu^(x) is continuous in (— l,l)x£?± (8.4) 

and the equality in (8.2) holds. Moreover, from (8.3) and the strong convergence of Vaij, to Vvt , 
we infer that 

/ A to [\7v to ,\7z]dx+ I A to [\7u to ,\7z]dx = for every z G L ~' 2 (U\K; dU). 
Ju Ju 
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Using this equation and arguing as before, we obtain 

which yields, in particular, the continuity of the map (t, x) i— » vf(x). Together with (8.4), this 
implies that the map (t, a;) i— » u^Oe) belongs to C 1 ((— 1, l)x5±) , which is equivalent to the second 
part of (8.2). Finally, the C°° regularity can be obtained by iterating the arguments above. □ 

The content of the next lemma is a pair of preliminary identities, which will be needed in the 
proof of Lemma 3.8. 

Lemma 8.2. Under the assumptions of Theorem 3.6, the following equalities hold on T: 

DX[X\\v] = -B[Xll,XH] +XW ■ V T {X-v), (8.5) 
§- t {v* t o$ t )| t=0 = DX[v,v\v— {DX) T [v\ = -(D T X) T [v]. (8.6) 

Proof. As X\\-V T (X-v) = X^-W(X-u) = {DX) T [v, X^] + (Du) T [X, X^] , identity (8.5) follows 
by observing that {Dv) T = (L>r^) T = B on T and that T X T is invariant for Dyv{x) for every 

x e r. 

Setting w t := {D<S> t )- T [v\ , it follows from (3.10) that 

Q 

^{v* t o$ t )\ t ^ = -{v-w)v + w. (8.7) 

The equality (8.6) then follows from (8.7) and the fact that w = -{DX) T [v\ . □ 

We conclude this appendix with the proof of Lemma 3.8. 

Proof of Lemma 3. 8. To simplify the notation in the sequel we will write simply u instead of . 
For igT let Ti(y), . . . , tn-i(v) denote an orthonormal basis of T y T which varies smoothly with 
y in a neighbourhood of x. For i e {1, . . . , N — 1} we have 

d Ti (d Ti u) = d Ti (Vu • n) — V 2 u [n, r»] + V« • d Ti n. 

Expressing d Ti Ti in the basis {r x , . . . , rjv-i, and using the fact that <9„w = on T, we obtain 

JV-l 

d Ti (d Ti u) = V 2 ?i [ri, + ^ (r fe • d Ti Ti)d Tk u. 

fc=i 

Hence, as rfc • <9 Ti Ti = — r, • <9 ri Tfc and n -Tk ~ for k ^ i, we have 

JV-l JV-l JV-l 

^ V 2 u [n, = ^ (rj • r fe )9 ri (<9 Tfc u) + d n T k )d Tk u 

i—l i,k=l i,k=l 

JV-l JV— 1 

i=l k=l 

Since u is harmonic, the first term in the previous identity coincides with — V 2 u [v, v\ , so that (a) 
follows. 

By differentiating along the direction r, the identity d v u — we deduce 

= d Ti {Vu ■ v) = V 2 u [n, v\ + Vu ■ d Ti v = V 2 u [r i; v] + V r w ■ d Ti u. 

Since d Ti v — Br^ and B is symmetric, the previous equality yields 

V 2 u [n,v] = -B[Vrw, Tj] for i = 1, ... ,7V - 1. 

By linearity the identity continues to hold if rj is replaced by any tangent vector. Hence, writing 
V 2 u [X, v] = (X ■ v) V 2 u[v, v\ + V 2 u [XW,v] and applying (a), we have 

V 2 u[A>] = -(X-u)A r u-B[V r u,X^ = -(X-v)A r u-B[V r u,X}, (8.8) 

where in the last equality we used the fact that B[Vrw] is tangent to T. This proves (b). We 
also note that identity (8.8) still holds when X is replaced by Vr« (in fact by any vector field), 
so that we obtain (e). 
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Using (b) and recalling that B = D T v , we find 
div r [(A • v)\7 r u] = (D r X) T [v, \7 r u] + B[X, V r «] + (X ■ v)A r u = (D T X) T [v, V r u] - V 2 u [X, u], 
which shows (c). 

Since Dv coincides with the Hessian of the signed distance function, we have by [1, Theorem 3, 
Part I] that 

d v (Dv) = -{Dv) 2 . 

Since H = drvv and Dv = D^v = B is symmetric on T, we immediately deduce (d). 
As Jj(^$ t ° §t)\t=o = v + Dv[X] , wc obtain (f) by comparison with (8.6). 
Finally, as Jj(J$ t )|t=o = divpA (see [15, Lemma 2.49]), we have by (8.6) 

f ($ t -("*t° )lt=o = Z-v + X-§- t (v iS>t o^> t )\ t=0 + {X-v)Aw r X 

= Z-v+(X- v)DX[v, v] - (DX) T [v, X] + (X • ^)div r A 

= Z-v- (DX) T [v,X n ] + (X-v)divrX 

= Z -v - DX[X\v] -X 11 -V r (X ■ v) + div r ({X ■ v)X) . 
Using (8.5) we obtain identity (g). □ 
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